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BACKGROUND OF THE INVENTION 

Field of t he I n ve n tio n 

The invention relates generally to ultrasonic and 
radiographic medical imaging, and particularly to three- 
dimensional, ultrasonic mammography for the purpose of 
detecting and visualizing breast lesions. 

Description of the Related Art 

The standard of care for managing breast disease 
includes (in order) detection, diagnostic work-up, 
definitive diagnosis, treatment, adjuvant therapy and 
follow-up. In a large number of cases, the health care 
provider will not consider a biopsy justified and rather 
will opt to follow the situation with examinations every 
three to six months. In other cases the health care 
provider will typically also want to follow the progress of 
the lesion after adjuvant therapies are administered. This 
implies repeated imaging of the region of tissue containing 
the lesion and comparison of the earlier and later images, 
to detect and characterize any changes in the lesion. 

Various imaging modalities are available to visualize 
lesions, including X-ray (most common) , ultrasonography, 
magnetic resonance imaging and other known methods . 
Whatever imaging method is used, to make meaningful 
before/after comparisons it is necessary to insure that the 
same region of tissue is under examination. Preferably the 
tissue should be projected from the same viewing angle in 
both before and after images . Comparison and even 
interpretation of images is complicated in part because 
the region of interest is a three-dimensional volume, while 
the image is a projection onto a two-dimensional plane or 
planes. Three dimensional positions and relationships must 



be inferred or calculated, based upon the available 
projections. Additional difficulties result because the 
breast may move slightly or be slightly deformed between 
the earlier and later images. Even slight misalignment 
5 makes it difficult to determine whether the same lesion is 

comparable in earlier and later images. Even if positions, 
compressions and projection angle correspond well in before 
and after images, side-by- side comparison of complex tissue 
patterns is difficult, and slight changes in tissue may 
10 escape detection. 

Summary of the Invention 

An image processing system and method visually 
documents historical cnanges in a medical image by 
15 processing and comparing historical with later images. A 

composite image is created which visually emphasizes 
differences between historical and later images. 
Preferably three-dimensional, ^digitized images, displayable 
in various projections, are stor^i for archival purposes on 
20 computer readable media. 

To properly relate\the historical and later images, an 
image processor preferably exploits an optical correlator 
to register the images accurately, by finding a 
transformation which produces a pre-determined degree of 
25 correlation between the images, then adjusting one image 

accordingly. The images a^e then compared, preferably 
volume element by volume element ("voxel by voxel") , to 
detect differences between historical and later images. A 
composite image is displayed with\synthetic colors or other 
3 0 visual clues to emphasize apparent changes in the imaged 

tissue, based on differences between the historical and 
later images . 

3yrief Pescription of the Drawings 
3 5 FIG. 1 is a system level block diagram of an apparatus 

in accordance with the invention; 

FIG. 2a is a summary level flow diagram of a method 
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for comparing historical with more recent medical imagery 
in accordance with; the invention; 

FIG. 2b is a flow diagram showing the continuation of 
the method of FIG. 2a; 

FIG. 3 is a perspective view of one geometric 
arrangement which can be used to obtain ultrasonographic 
imagery of a human breast for use by the invention; 

FIG. 4 is a perspective view of the arrangement of 
FIG. 3, showing further details of a scanning scheme for 
obtaining three-dimensional image data from a human breast; 

FIG. 5a is a flow chart showing the initial steps of 
a method of registering earlier with later images, suitable 
for use in the registration step of FIG. 2a; 

FIG. 5b is a flow chart continuing from FIG. 5a, 
showing the further steps in the registration method; 

FIGs. 6a, 6b, and 6c, show simplified examples of an 
input image, a filter template, and a resulting correlation 
output image, respectively, in an example of a suitable 
cross -correlation operation which discovers positional 
offset of correlated images in the method of FIGs. 5a and 
5b; 

FIG. 7 is a perspective view of an arrangement of 
planar slices of a breast which can be selected for image 
processing of three-dimensional breast images by the 
invention; 

FIG. 8 is a perspective diagram representing a typical 
visual display of a composite image created by the 
invention to document historical changes in breast tissue; 
and 

FIG. 9 is a symbolic diagram of an optical correlator, 
optionally used in accordance with a variation of the 
invention for rapidly performing two-dimensional 
correlation operations, thereby aiding in registering the 
historical and later images. 



Detailed Description of the Invention 

Overview of. Apparatus 

FIG . 1 shows an apparatus in accordance with the 
invention and suitable for practicing its method. An 
ultrasonographic imaging system 20 (or its equivalent) 
images a patient's breast 22. The imaging system 20 
provides digital image data to an image processor 24 via an 
input channel 26. An optical correlator 30 is preferably 
interfaced with and controlled by the image processor 24 to 
provide high-speed image processing (correlations) of pre- 
processed image data. A user input device 32 (typically a 
keyboard and/or graphical pointing device such as a mouse) 
is interfaced to the image processor 24 to allow user 
control of the image processor 24 . Graphic output is 
displayed by the image processor 24 on a display 34, which 
is preferably a color- capable video display. A printer 36 
is preferably also interfaced with image processor 24 to 
produce "hard copy" printouts which record the imagery, most 
preferably with multi-color, high resolution graphics. A 
storage device 38 such as a CD-ROM writer, digital tape 
storage, DVD, or similar digital storage device should 
preferably be also interfaced with the image processor 24 
to record and store imagery in a digital data format, for 
archiving. Optionally, the image processor 24 could be 
also interfaced to a data network 39 to allow the exchange 
of data with distant users, or to access other sources of 
image data. Instead of or in addition to the data network, 
the image processor 24 can be interfaced with a digitizing 
scanner 40 which allows input of historical imagery such as 
X-ray mammograms. 

The image processor 24 is preferably a 64 bit 
workstation such as the Silicon Graphics 02, although less 
powerful processors could be used at the expense of speed 
or resolution. The ultrasonographic imaging system 20 
should preferably be capable of sweep scanning, to provide 
the multiple slices of imagery for assembly into a three- 
dimensional data set. Either a dedicated processor 



integral with the ultrasonographic imaging system 2 0 could 
provide such assembly, or it could alternatively be 
performed by the image processor 24. 

5 Overview of Method . - 

FIGs . 2a and 2b summarize a method used in accordance 
with the invention to compare historical with later, 
mammographic images. First, the image processor 24 inputs 
(step 50) historical image data. Preferably the image data 

10 is available from a data network 39, but alternatively it 

could be obtained by scanning film imagery by digitizing 
scanner 40. If available, three dimensional historical 
image data is preferred. As another option, the historical 
data could be retrieved from a mass storage device 3 8 

15 rather than loading by inputting the data. 

Next, in step \52, pathological feature data is 
preferably loaded by \he image processor 24 (preferably 
from a data network bu\ alternatively by retrieval from 
storage or through another input channel) . The 

20 pathological feature data\is useful for narrowing the 

region of interest for imkge processing, and could be 
provided in any of several useful forms. Typically, a 
physical region of interest (KOI) in the historical image 
has been previously identified Nby a radiologist or other 

25 professional, and the coordinated of that region recorded 

in association with the historical, image. In that case, 
the coordinates of the ROI are th\ pathological feature 
data. Alternatively, certain \ image shapes or 

characteristics, stored in a pathological image library, 

3 0 could be compared with the currents, image by image 

correlation to define the pathological region as described 
below in connection with FIGs. 6a -61c. A further 
alternative is to employ known automated methods for the 
detection of abnormalities in digital mammograms. For 

35 example, see U.S. Patent No. 5,4,-91,627 to Zhang et al . 

(1996) . The locations of any detected suspect lesions, as 
determined by such an automated method, could bfe used as 
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the pathological feature data to identify, and locate the 
ROI in the image.. \ 

Next (step 54) a coarse resolution, three-dimensional 
image sweep of the patient's breast is taken, preferably by 
5 sweeping an ultrasonic imaging head linearly across the 

breast thereby obtaining a plurality of image slices which 
collectively define a three-dimensional image. The. 
resulting coarse image is stored (also in step 54) . The 
coarse breast image of step 54 is then manipulated by the 

10 image processor to properly register it (step 56) with the 

historical imagery, to at least a preliminary degree of 
registration. This initial, coarse level of registration 
can be performed by processing only a very simplified 
image, to improve processing speed. For example, the image 

15 can be first compared pixel by pixel (or voxel by voxel) to 

a threshold value, and a binary product image produced 
based on the result at each pixel (voxel) . In the 
threshold processed image, only the most obvious features, 
such as the breast outline and chest wall, will be 

20 recognizable. The images can then be rotated and shifted 

either manually (by operator command) or by the 
registration method described below in connection with 
FIGs. 5a and 5b, to crudely register the images. The 
initial registration of step 54 preferably operates on the 

25 whole image but at very low resolution: for example, the 

entire image is suitably mapped, after thresholding, onto 
a 256 by 256 matrix, then registered by the correlation 
method of FIGs . 5a and 5b, This suffices to approximately 
register the images, based \m gross features only. 

30 Next, to obtain improved registration between the 

historical and new image, the\image processor correlates 
the pathological feature set with the new image (step 58) , 
preferably by the method describedNpelow in connection with 
FIGs . 5a and 5b. The information gained from the 

35 correlation operations allows betters registration of the 

new image with the historical imagery arid defines an ROI in 
the new image. The ROI thus defined rs then re-scanned 
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(step 60), preferably as described above, with scanning 
ultrasound equipment r>ut with a finer level of resolution. 
Finer resolution can\ be obtained by using slower scan 
speeds, higher frequency ultrasound, finer and more 
numerous receive sensors, or other techniques as known in 
the art. The fine scan, being preferably thus restricted 
to the ROI, can be executed, in a shorter time than would be 
required to image the entire breast at a fine level of 
resolution. \ 

The image processor 24 next proceeds to register the 
new, finely resolved ROI image with the historical image of 
the corresponding ROI (step 62) . This new registration 
step 62 is performed with finer resolution than the 
previous coarse registration (step 56) , but preferably the 
operations are restricted to a smaller region of imagery, 
thus preserving a practical, limited computation time. This 
results in accurate and precise registration between the 
historical and the new tissue ROIs. Preferably, both ROIs 
are three-dimensional volume regions which are aligned by 
registration in three dimensions. 

After the historical and new images are aligned, they 
are compared in step OA on a point-by-point basis, to 
obtain a difference imagfe which emphasizes the temporal 
changes which have occurred\ For example, if digitized two 
dimensional images are considered, the images are 
preferably first normalized to\a common average value, then 
corresponding pixels are subtracted and the difference 
values stored in a corresponding^ position in a difference 
image. Analogously, if three-dimensional imagery is 
available (which is greatly preferred) , voxels at 
corresponding locations in the earlAand later images are 
subtracted one from the other, and thk result stored in a 
voxel of a difference (volume) image. "voxel" is a unit 
of graphic information that defines a small volume element 
in three-dimensional space. It is the tmree -dimensional 
analog of a "pixel" which defines an area element in two- 
dimensional space.) Thus, normalized befate and after 



images can be substractively compared, voxel -by- voxel , to 
obtain a three-dimensional difference image which 
represents the differences between the before and after 
images . \ 

As a result of step 64 , at least three images are 
available to the image processor 24: an early (historical) 
image; a current image; and a difference image. These 
images are then combined and/or selected for display (step 
66) in various ways, asv directed by user input from input 
device 32. Finally, the resulting image or images are 
displayed (step 68) and preferably stored (step 70) and/or 
exported (step 72, preferably via the data network ) . It is 
highly preferably that the\ imagery be coded to emphasize 
visual differences between Vhe earlier and later images. 
For example, the difference, image can be coded in a 
distinct color for display, then superimposed on the early 
image, the later image, or a\composite. Various other 
conventional methods could be \equivalently employed to 
visually emphasize image differences. For example, in one 
variation the difference image \ can be displayed in 
intermittent , flickering display superimposed on either of 
the historical or current images\. Significantly, the 
invention easily detects dif ferencesy between the earlier 
and later images, visually displays the images, and 
visually emphasizes these dif f erences . \ 

The product images from step 66 should preferably be 
recorded (step 72) for archival purposes, preferably on a 
high-density, long persistence medium such as magnetic tape 
memory or writable CD-ROM. Preferably a three-dimensional 
volume image should be stored, but to save storage space, 
operator selected projections can be derived (by 
programming the image processor 24 under control from the 
operator) and stored as two-dimensional imagery. Three 
dimensional imagery is preferred, despite its greater 
storage demands, because it can be more easily compared 
with other images (either previous history or subsequent 
development) . The three-dimensional imagery can be 
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digitally manipulated by rotation, scaling, translation 
then projected to provide comparison with any view; two 
dimensional imagery does not provide sufficient information 
to carry out the aforementioned manipulations without loss 
of information or introduction of ambiguities in the 
resulting image. The location of significant changes in the 
image should preferably be tagged with an identifier, to 
cross -link to a potential future tissue sample specimen 
number. 

The method may be repeated to compare multiple 
historical images.. Either one or multiple combined images 
can be stored for archival and to document historical 
developments . 

Registration method: 

The success of the s above described procedure depends 
in part upon the accuracy \with which the earlier and later 
images are registered (iA steps 56 and 62 of FIG. 2a) 
before comparison. Slight Viovement or deformation of the 
breast tissue or the scanning apparatus is to be expected 
between the earlier and later scans. The movement may 
include translation, rotation\ about any axis, or slight 
compression or expansion (in adaition to biological tissue 
changes) . To adequately register the images in step, 
therefore, a computationally practical and fast method of 
registration is preferred. A . ©referred method of 
registration takes advantage of the ^specif ic computational 
abilities of an optical correlator Mdiscussed in detail 
below in connection with FIG. 9) . Tnis preferred method 
(suitable for use in step 64 of FIG. 2a \ is best described 
with reference to an example of a particular coordinate 
system, to aid in visualization. \ 

FIG. 3 shows how a sonographic or radiographic breast 
image might be oriented with respect to a , particular useful 
coordinate system. The system is conveniently illustrated 
in a Cartesian, rectilinear coordinate system having 
linear, perpendicular axes x, y and z, but the invention 



is not limited to such a coordinate system. The patient ! s 
breast 22 is scanned with ultrasound by the ultrasonic 
imaging system 20 (omitted for clarity, but shown in FIG. 
1) . With a patient preferably sitting facing the imaging 
system, the patient ! s breast 22 is preferably slightly 
compressed between pressure plates 83 (upper) and 84 
(lower) in a manner that makes maximal use of prior 
information (such as x-ray images) . In positioning the 
breast between the pressure plates, edges of the plates 
will contact the patient \s chest above and below the 
breast. Because of slight movement between image 
acquisitions, the patient's breast in the imaging system, 
the axes of the later imagery do not in the general case 
exactly correspond with the x, y and z axes of the 
historical image, but may differ by a coordinate 
transformation: for example, they may differ by translation 
in the x, y or z directions, and by rotation about any 
axis. Rotation about the z axis is especially likely. 

With the breast 22 in position, ultrasonic scanning is 
preferably performed in planes (or surfaces) , which will in 
general be non-parallel to that of the Y-Z plane in FIG. 
3. FIG. 4 shows a typical geometry in which the scanhead 
9 0 includes a linear array of ultrasonic transducers 
aligned parallel to the y axis. The scanhead 90 transmits 
ultrasonic pulses in the directions of the parallel lines 
96, which are preferably perpendicular to the x'-y' plane 
and parallel to the z'-y' plane. The array of transducers 
in scanhead 90 probe the underlying tissue lying 
(approximately) on lines 96 by detecting returns of the 
ultrasonic pulses caused by acoustic impedance 
discontinuities or reflecting surfaces within the tissue. 
The delay time between transmitting a pulse and receiving 
an return is indicative of the depth of the discontinuity 
or surface which caused the return. A characteristic such 
as magnitude, phase, or frequency of the returns is 
digitized and is plotted against the depth (z 1 axis) 
information and the information from the multiple 



transducers (dispersed in the y' direction) is assembled to 
construct an array representing a cross-sectional view of 
the tissue in a slice 98 parallel, to the y'-z' plane and 
lying under scanhead 90 . 

Multiple slices can be scanned either by providing 
multiple scanheads, a two -dimensional scanhead. array, or by 
moving the scanhead across the breast, for example in the 
x 1 direction in FIG. 4. The planes of only a few of the 
multiple slices, specifically slices 98, 100, and 102, are 
shown. In practice a large number of slices is desirable, 
for better resolution. A complete set of such slices is 
preferably scanned to form a three dimensional information 
set for at least some region of interest (ROD chosen from 
the breast, which is preferably stored in a data structure 
(such as a three-dimensional array) to represent a three- 
dimensional image. 

Ultrasonographic equipment is available commercially 
which can be used as the ultrasonographic imaging system 2 0 
described above. A two-dimensional array of 

ultrasonographic data is usable by the invention, but with 
a diminished amount of useful information in the resulting 
display. 

Keeping in mind the exemplary coordinate system of 
FIG. 3, a method suitable for registering the historical 
and later images (step 64) is shown in the flow chart of 
FIG. 5a (and continued on FIG. 5b) . By this procedure the 
image processor 24 determines the proper coordinate 
transformations of scale, position, and rotation which will 
align the historical and later images. The image processor 
24 accesses (step 102) the stored ultrasound images from 
ultrasonographic imaging system 20 and extracts (step 104) 
a two-dimensional representation preferably by projecting 
or "collapsing" the three-dimensional ultrasound data onto 
a single plane. One method of doing this is by "cumulative 
projection": a projection of the three-dimensional data set 
onto a two-dimensional plane by summing the data entries 
along vectors which, are perpendicular to the projection 
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plane. One such vector, vector 106, is indicated on FIG. 
4 for illustration. The density values associated with the 
voxels (three dimensional discrete volume cells) such as 
voxel 110 are summed along the vector 106. The summation of 
5 those density, values yields a scalar value which indicates 

the sum of the tissue densities along that vector. This 
scalar value is associated with the pixel 112 at the 
intersection of the vector 106 with the x'-y 1 plane. 
Repeating this summation for multiple parallel vectors 
10 results in a set of values which defines the projection of 

the three-dimensional sonographic imagery onto the x'-y ! 
plane. This projection is preferably applied to both the 
historical and later imagery. Thus, returning to FIG. 5a, 
both three-dimensional data sets are projected onto 
15 respective two-dimensional images and stored (step 114) . 

The purpose of collapsing the three-dimensional images 
into two dimensions is to facilitate correlation, which is 
preferably accomplished by use of an optical correlator. 
Such correlators conventionally require two-dimensional 
20 imagery as correlation inputs. In some applications, the 

historical imagery will already be in two-dimensional 
projection, having been previously selected and projected 
to adequately visualize the lesion (s) of interest. In such 
a case only the later imagery must be collapsed. 
25 As an alternative collapsing the images by simple 

projection, as described Vbove, a three-dimensional image 
can be processed to created a "discriminate filter." Such 
filters are known and facilitate registration by providing 
a correlation filter which a composite of many images which 
30 differ by rotation about an axrs. This allows testing of 

multiple, rotated slices simultaneously, by correlation 
with a composite discriminate filter. Once a discriminate 
filter is determined with acceptable\correlation (at least 
equa to a pre-defined level of correlation) an individual 
35 filter is then selected from the set df slices which was 

used to compose the composite. In some cases, this method 
can greatly accelerate the search for bestVcorrelation. 
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The projected images are optionally further pre- 
processed (step 116) . Pre-processing 116 can include any 
of a variety of known image processing techniques including 
(without limitation). : contrast modification, smoothing, 
geometric transformation, thresholding or the selection of 
a region of interest. Depending on the type of optical 
correlator 30 used, as discussed in detail below, this step 
may also include two-dimensional Fourier transformation of 
the digitized x-ray image to prepare it for subsequent 
optical correlation in a Vanderlugt optical correlator. 

Next the image processor 24 adjusts (step 118) the 
relative scale of the two images so that they better 
correspond in scale. This can be done by various methods. 
For example, one method is to match the total area of the 
cross section of the breast area between the outer outline 
and the chest wall in both images. In this method, the 
images should preferably first be processed to remove low 
contrast features, leaving only the easily visible outline 
of the breast and the chest wall. The area between these 
features in the two dimensional images is then measured, 
for example by numerical integration by the image processor 
24. The area should correspond in both images. If the 
areas do not correspond, it is possible that the degree of 
compression has changed, which can compress or expand the 
breast. scaling correction factor is then preferably 

applied in step 118 to correct as closely as possible. On 
the other hand, it is possible in many cases to maintain a 
relatively constant compression. In such cases, little or 
no re- scaling is necessary. 

After scale correction^ the image processor 24 
determines the rotation and translation necessary to align 
the images, preferably by interactively performing the 
steps grouped within the instruction loop 122 in FIG. 5a. 
First, two variables are initialized (step 128) : a counter 
j to control execution of an instruction loop, and an 
associated rotation angle otj . Next, a cross -correlation of 
the dual images is computed (step 130) . Preferably this 
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step is accomplished by using the optical correlator 30 to 
perform the actual correlation computation, under control 
of the image processor 24 . (The details of the optical 
correlator are discussed below, in connection with FIGs . 5 
5 and 6) The cross-correlation (step 130) produces a two- 

dimensional correlation output image indicating the degree 
of image cross-correlation, which is stored (step 132) 
along with the associated rotation angle otj . The image 
processor then checks (step 134) the counter variable to 
10 discover whether it has completed a prescribed number of 

iterations of the instruction loop 122 . 

Next, if the counter variable j has not reached jmax, 
the image processor 24 continues and rotates (step 13 6) 
one of the dual images relative to the other by some 
15 angular increment, for example by rotating the historical 

image one degree about an axis centered in the frame and 
parallel to the z axis. The counter is incremented (step 
13 8) and the procedure loops back to step 13 0 to perform 
Sjj another cross-correlation, this time with the images 

" 2 0 rotated incrementally. The procedure is repeated until 

q some number (jmax) of differently rotated correlations has 

Sh been performed. The parameter jmax should be chosen to be 

Jr! large enough so that the range of the associated angle o£j 

Q encompasses the anticipated maximum possible range of 

w 25 rotation. For breast examination in the geometry shown in 

FIGs . 3 and 4, a range of less than 10 degrees is adequate 
in most cases . 

The rotations applied in step 13 6 are not limited to 
rotations about a single axis, but can include rotations 
30 about multiple independent axes (or, equivalently rotations 

about an axis obliquely oriented with respect to the 
orthogonal axes shown) . This allows the correction for an 
oblique viewing angle of one image with respect to the 
viewing angle of the other. 
35 After the counter j reaches jmax the image processor 

24 exits the instruction loop 122. The procedure continues 
as diagramed in FIG. 5b. The correlation output images 
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previously stored in the various iterations of step 13 2 are 
compared (step 140) with one another to find the 
correlation output image with the maximum correlation, and 
its associated angle of rotation oCj . The value oc m (the 
rotation angle which produced maximum correlation) is 
stored (step 142) and either the historical or the later 
image is rotated (step 144) by a m to bring it into the 
same orientation as its counterpart . 

It should be understood that in addition to 
discovering the correct scale and rotation angles, the 
cross-correlation (step 130) in the above described 
procedure produces an output image which reveals the 
correct translation (position shift) which best aligns the 
dual images. The translation information is contained in 
the result of the cross-correlation operation (preferably 
in two-dimensions) defined as: 

H(x 9 y) = g{x, y) * f(x, y) = fi f(a,/3)g{x -a 9 y- p)dadp ( 1 ) 

Where f and g are functions of two variables (images) , x 
and y are the spatial variables of the two-dimensional 
images, a, and (3 are dummy variables of integration, H is 
the cross -correlation of functions f and g, and the range 
of integration is across the entire image. If f and g 
differ only by a positional offset in x and y, H(x,y) will 
have a sharp peak at a position x p ,y p which is displaced 
from a central, correlation alignment position (typically 
defined as x=0, y=0) by a displacement corresponding to the 
offset between f and g. This well known result has been 
used to recognize and locate features of initially unknown 
locations in a field by reference to a template. See, for 
example, Russ, John C, The Image Processing Handbook (CRC 
Press, 1992), pp. 218-24. A simplified example is shown in 
FIGs. 6a- 6c. The input image, a star 146 at an offset 
position shown in FIG. 6a, is correlated with a filter 
derived from the centered star- shaped template 147 shown in 
FIG. 6b. The resulting correlation output shown in FIG. 6c 
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has a peak 148 at a position x p/ y p corresponding to the 
offset between the input image and the filter template. To 
align the images with a correlation peak at position x p ,y p 
it is sufficient merely to translate one of the images by 
a displacement equal to the offset x p/ y p . 

Returning to FIG. 5b, the image processor 24 next 
analyzes (step 154) the correlation output image to locate 
the positional offset of the correlation peak from an 
aligned correlation position, then translates (step 156) 
one image relative to the other as necessary to better 
align the images. After finding the optimal combination of 
rotations, scalings and translations to align the earlier 
and later images, the image processor 24 preferably stores 

(step 158) the transformed images and the transformation 
parameters in its associated memory and preferably outputs 

(step 160) the transformed images to a display device 34. 

In one embodiment the invention utilizes additional 
three-dimensional information about the subject body by 
further correlating the images in order to align the z 
direction (depth) . To accomplish this, the 

ultrasonographic imagery is first partitioned by image 
processor 24 into conveniently defined slices, for example 
slice 164 as shown in FIG. 7. Each slice includes one or 
more layers of the three dimensional ultrasonographic image 
data. The slices are defined and calculated by image 
processor 24, for example by summing data points along 
vertical vectors such as 165, to collapse multiple thin 
layers into a thicker slice (a "partial cumulative 
projection") . In FIG. 7, for example, multiple thin layers 
166 of ultrasonographic imagery (shown only partially to 
clarify FIG. 7) might lie between bottom slice 164 and the 
immediately overlying slice 167. The partial cumulative 
projection is taken point -by-point , by summing the 
ultrasonographic image values at points along vectors such 
as vector 165, and accumulating the result to the point 168 
where vector 165 intersects the slice 164. The accumulated 
data values at each defined point on slice 164 collectively 



define the slice. 

In a typical application, slices with thickness of 5 
millimeters or less are suitable. Although planar slices 
are convenient for purposes of illustration, in some 
5 applications the slices might usefully be taken along non- 

planar contours. Such slices are also within the scope of 
the invention. Thinner slices are desirable for better 
depth definition with thin features . 

To best align the earlier and later images, 
10 corresponding slices in each are individually correlated to 

find their best registration, for example by the method 
described above. In one variation of this technique, the 
slices are defined parallel to the X f Y ! plane. In another 
variation, the slices are defined in another plane (the 
15 slice under the scanhead, parallel to the Z ! Y f plane, for 

example) and registered by the above described method, then 
reassembled. By registering each slice, the three- 
jhy dimensional imagery can be aligned to account for 

variations in shear or torque between the earlier and later 
20 image acquisitions, as might be caused by variable 

deformation of the breast under varying pressure. 

The fine registration step of step 62 (in FIG. 2) is 
preferably performed by essentially the same method as the 
coarse procedure: i.e., the method of FIGs . 5a and 5b. The 
25 fine registration will yield correlation results which rely 

upon finer features than the coarse registration. Small 
constellations of microcalcif ications, in distinctive 
patterns, will yield correlation results which allow 
registration of the ROIs at a very fine scale. However, 
30 perfect correlation will not be obtained in general, 

particularly if tissue changes have occurred between the 
earlier and later images. Nevertheless, detection of 
optimum correlation should permit satisfactory 
registration. In those cases in which a dramatic tumor 
35 growth has occurred, the tissue changes will be more 

dramatically visually conspicuous, even with imperfect 
registration; contrariwise, if only slight tissue changes 
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have occurred, good correlations will be obtainable, 
resulting in precise alignment. 

After the earlier and later ROI images are adjusted to 
best register them, theyV can be either combined by adding, 
voxel -by -voxel, or compared by subtracting, voxel - by - voxe 1 . 
Differences between the images are easily highlighted by 
changing the color at voxels which show high variation 
between the dual images . BIG . 8 shows an example of one 
typical display of a combined image (in this case, a 
composite of earlier and latter images, with differences 
highlighted) . The outline of the breast 22 is shown, with 
highlighted suspected lesiom 200 (simplified for 
illustration) as revealed by a recent ultrasonogram. 
Other more static regions of density 202 and 2 03 are shown, 
which would suitably be displayed\in a neutral color or 
grey scale. The earlier lesion region 206 is surrounded by 
the later, larger lesion 200. Suitably the later region 
200 would be highlighted, by example by display in a 
conspicuous color such as pink. The Waller lesion 206 
(visible in the earlier historical image\ would preferably 
be coded in a different color (for example, green) . This 
three dimensional image, easily digit ized\ and archived on 
computer readable media, provides easily readable 
documentation of the historical image changes, and can be 
retrieved for evidence of tissue changes A The image 
processor can also extract slices such as that cut by 
imaginary plane 302, and display the slice \as a two- 
dimensional section for detailed inspection. Preferably, 
the processor is programmed to respond to user \ input so 
that any slice can be selected for display, or the user can 
sweep through multiple slices to view details of interest . 

In the procedures depicted in FIGs . 5a and 5b it is 
highly preferable that the correlation operations be 
carried out by an optical correlator. In the preferred 
embodiment, the image processor 24 electronically writes 
the dual images to the optical correlator 30. The optical 
correlator 30 preferably performs the correlation 
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operations and returns a resulting correlation image to the 
image processor 24. 

Optical correlators use wave optics to correlate 
images in two dimensions by first performing essentially a 
two-dimensional spatial Fourier transform on a two- 
dimensional source image. This method takes advantage of 
a well known mathematical property of the Fourier 
transform: many operations including correlation are more 
easily calculated in the Fourier transform domain than in 
the original spatial domain. Specifically, a two- 

dimensional correlation operation is defined by equation 1 
(above), where f(x,y) and g(x,y) are the two-dimensional 
functions or images to be cross -correlated, and a and (5 are 
dummy variables of integration. This operation can be 
performed digitally for each point x,y by numerical 
techniques, but a very large number of calculations are 
required even for one image correlation. Performing such 
an operation digitally is very time consuming and requires 
inconvenient lengths of time on any but the fastest digital 
computers . 

Unlike a conventional digital computer, an optical 
correlator can very rapidly perform a correlation 
operation, correlating a source image with a filter image 
by (1) optically Fourier transforming a source image, (2) 
comparing the source and filter image in the Fourier 
transform domain, and then (3) performing an inverse 
Fourier transformation to produce the correlation pattern 
in a spatial representation. An optical correlator can 
accomplish these operations much faster that a digital 
computer because the optical Fourier transformation is 
executed as a simultaneous operation on all points of the 
source image, using inherent properties of wave optics to 
generate the Fourier transform in two dimensions. The 
speed of the device is limited for practical purposes only 
by the available read and write speed of the data transfer 
to the correlator; the actual optical processes occur in 
fractions of a nanosecond in typical optical correlators. 




The principles of the optical correlator are known, 
and have been described for example in the U.S. Patent No. 
5,311,359, to Lucas et al . Compact optical correlators 
suitable for use in the present invention are commercially 
5 available from Litton Data Systems, Inc., in Agoura Hills, 

California, as well as from other sources. Alternate types 
of optical correlators such as the Joint Transform 
Correlators described in U.S. Patent No. 5,650,855 to 
Kirsch et al . , U.S. Patent No. 5,216,541 to Taksue et al . 
10 or U.S. Patent No. 5,438,632 to Horner, may also be 

employed with the invention. 

For purposes of describing the present invention, the 
optical correlator 30 may be considered functionally as an 
electro-optical device having three (electronic) ports, as 
15 shown in FIG. 9. The three ports include: (1) an image 

m input port 210 for receiving an electronic signal encoding 

Q an input image for correlation,* (2) a filter input port 212 

h"i for receiving a second electronic signal encoding a second 

ifl image or "filter" for correlation; and (3) an output port 

f 20 214, typically from a charge coupled device (CCD) imager, 

Q which converts the correlation image into electrical form 

Zl for output. In addition the device requires a source (not 

ry 

jy shown) of preferably coherent electromagnetic radiation, 

P typically a laser, which provides the medium used for 

^ 25 computation. 

Both the image input port 210 and the filter input 
port 212 are realized as two-dimensional spatial light 
modulators (SLMs) organized as two-dimensional image 
matrices, with addressable image pixels (typically arranged 
3 0 in the familiar row and column pattern) . Accordingly, the 

input image must be formatted (suitably by image processor 
24) to fit the matrix; and each pixel of data should 
preferably be addressed, under control of the image 
processor 24, to the spatially corresponding pixel on the 
35 SUM. For example, in one embodiment of the invention, the 

image input port and the filter input port are realized as 
256 x 256 pixilated matrices. Accordingly, in this 
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embodiment the image processor 24, as part of pre- 
processing step 116 (in FIG. 5a) , maps an ultrasonographic 
image onto a 256 x 256 matrix for output to the optical 
correlator 30. In a typical embodiment of the 

5 invention a Vanderlugt type optical correlator is used. In 

such a correlator the "filter" image must be pre-processed 
by two-dimensional Fourier transformation. In such an 
embodiment the image written to the filter port is 
preferably Fourier transformed by image processor 24 (for 
10 example in pre-processing step 116) , to provide a frequency 

domain pattern. In an alternate embodiment, a joint 
transform correlator may be used as optical correlator 30. 
This eliminates the need for the digital Fourier 
transformation of the filter image, as the transformation 
15 is optically performed by the joint transform correlator. 

Note that in conventional, practical high speed 
optical correlators, the SLMs and the photodetector matrix 
consist of discrete pixels rather than a continuously 
modulatable surface. Thus, the Fourier transformation and 
2 0 correlation operations can only approximate the discrete 

cross-correlation (given by equation 1, above) . However, 
the approximation obtained is sufficient for image 
processing for most applications. 

When the input the fxSLter images have been written to 
the input and filter ports 210 and 212, the optical 
correlator produces an output image which is a two 
dimensional output correlation pattern having an optical 
peak or peaks (bright spot) at the position of greatest 
correlation between the collapsed sonographic image and the 
30 radiographic image. The degree ofy correlation is indicated 

by the intensity of the output signkl. The position of the 
output peak on the two-dimensional matrix of the correlator 
output CCD indicates the translations or shifts of the 
images relative to one another. The. output image is read 
35 from the output photodetector (CCD) \ 214 by the image 

processor 24 in the conventional manner, typically by 
shifting the CCD voltage values out sequentially in rows 
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(or columns) and then\digitizing the output levels. 

In many applications in which two images are to be 
correlated, it is irrelevant which image provides the 
filter and which the corresponding input image, provided 
5 that the filter image is presented (in a Vanderlugt type 

correlator) in a Fourier transform representation. 
However, in some particular embodiments it may be 
preferable to route the historical imagery to the input 
port 210 rather than the filter port 212. For example, if 
10 discriminate filters are derived from the later ultrasound 

imagery and are to be correlated, these filters would 
usually be derived in a Frequency domain representation and 
would thus more appropriately routed to the Filter port 
212. On the other hand, if a search for a correlation is 
*jf 15 to be done with a brute- force, match search, then the 

EH multiple projections derived from the later imagery should 

jf; be applied to the input port 210 (thus avoiding the need to 

ru perform numerous two-dimensional Fourier transformations) . 

~ ! Although the invention is described in terms of linear 

g 20 transformations of the coordinates, such as translation, 

O rotation, and scale multiplication, the invention is not 

-J limited to linear transformations. Non-linear 

Ly transformations of coordinate systems may be useful in some 

^ applications. For example, the ultrasonographic 

"~ 25 information may be obtained with the breast differently 

deformed, as by a change of position of the subject, or by 
instrument pressure. By applying a mathematical 

transformation, which may in general be non- linear, a 
better mapping of the deformed subject breast onto the 
30 original subject breast can be obtained. Similarly, some 

scanning techniques may involve curvilinear, non-cartesian 
coordinate systems which would be treated with non- linear 
transformations. 

While illustrative embodiments of the invention are 
35 described above, it will be evident to one skilled in the 

art that numerous variations, modifications and additional 
embodiments may be made without departing from the 
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invention. For example, the construction of the 

ultrasound imaging system or the geometries and coordinate 
systems employed may be varied. Various means of data 

storage, transmission, or processing may be employed. The 
resolution or type of image that is sent from the image 
processor to the optical correlator could also be altered. 
Three-dimensional cross -correlations are also possible (but 
computationally complex) . To the extent that such 
operations can be decomposed into multiple planar 
operations, the use of the optical correlator as described 
above could greatly accelerate computations. Accordingly, 
it is intended that the invention be limited only in terms 
of the appended claims . 



